Emergency department performance assessment using administrative data: A managerial framework

Administrative data play an important role in performance monitoring of healthcare providers. Nonetheless, little attention has been given so far to the emergency department (ED) evaluation. In addition, most of existing research focuses on a single core ED function, such as treatment or triage, thus providing a limited picture of performance. The goal of this study is to harness the value of routinely produced records proposing a framework for multidimensional performance evaluation of EDs able to support internal decision stakeholders in managing operations. Starting with the overview of administrative data, and the definition of the desired framework’s characteristics from the perspective of decision stakeholders, a review of the academic literature on ED performance measures and indicators is conducted. A performance measurement framework is designed using 224 ED performance metrics (measures and indicators) satisfying established selection criteria. Real-world feedback on the framework is obtained through expert interviews. Metrics in the proposed ED performance measurement framework are arranged along three dimensions: performance (quality of care, time-efficiency, throughput), analysis unit (physician, disease etc.), and time-period (quarter, year, etc.). The framework has been judged as “clear and intuitive”, “useful for planning”, able to “reveal inefficiencies in care process” and “transform existing data into decision support information” by the key ED decision stakeholders of a teaching hospital. Administrative data can be a new cornerstone for health care operation management. A framework of ED-specific indicators based on administrative data enables multi-dimensional performance assessment in a timely and cost-effective manner, an essential requirement for nowadays resource-constrained hospitals. Moreover, such a framework can support different stakeholders’ decision making as it allows the creation of a customized metrics sets for performance analysis with the desired granularity.


Introduction
Overcrowding and, consequently, long length of stay (LLOS) are two of the starkest examples of poor performance in emergency departments (EDs).According to major media companies, as in mandatory reporting [22].Thus, routine data represent an important source for performance monitoring at different levels, such as macro (national, regional, local) or micro (individual provider, clinical area, etc.).
The goal of this study is to harness the value of routinely produced administrative records by developing a framework for multidimensional performance evaluation of EDs that would support internal decision stakeholders in managing operations.This work responds to the need of a practical tool to be used by ED stakeholders for support in decision-making, allowing for prompt problems detection in care quality and care delivery based on already available information.To the best of the authors' knowledge, the only study offering a practical framework for ED performance management has been conducted by Nu ´ñez et al. [23].The authors proposed a set of 75 performance indicators related to processes carried out by EDs that are relevant for monitoring purposes.However, one serious limitation to the Nu ´ñez et al. [23] framework's application is that the data is uses are not available, but need to be purposely collected, thus adding an additional burden on an already resource-constrained system.The framework developed in this study, also validated by a series of interviews with a panel of experts, overcomes this limitation as it uses data that are collected on a daily basis by hospitals' Information Systems.

Methods
In this research, no administrative record related to any patient has been examined, cited, nor referred to.Data in this article consists of theoretical indicators extracted from published scientific papers (properly cited).On the other hand, for the interviews evaluating the framework, i.e., its indicators, doctors and managers provided their verbal consent.Anyway, the Ethical Committee expressed a favorable opinion on the diffusion of the contents in this manuscript.
In detail, the research method is comprised of two broad phases: 1) the design of a multidimensional framework to harness hospital administrative data; 2) an expert session to evaluate the framework.This method loosely follows Gu and Itoh [24], Mantwill, Monestel-Umaña, and Schulz [25], and Stremersch and Van Dyck [26].
In the first phase, the framework is designed bottom up using the logic suggested by Keegan, Eiler, and Jones [27], which includes a definition of the strategic objectives and a decision of what to measure to reach such objectives.In practice, the following steps have been performed: 1. definition of administrative data and overview of the information they contain; 2. definition of the framework's characteristics from the perspective of ED decision stakeholders, and the corresponding key attributes of measures and indicators; 3. review of the academic literature on ED performance measures and indicators; 4. selection of metrics to include in the framework based on the characteristics defined in step 2); 5. organization of the selected indicators into a framework.
In step 4), particular emphasis has been given to the selection of measures that are easy to calculate using information already available in ED administrative data sources.This is to avoid that the managerial framework be yet another burden causing information overload and poor performance [24].
Regarding the second phase, four stakeholders have been identified for the semi-structured interviews, which took place in May 2022.They represent target users of the framework with different roles [28].This role-wise classification reflects diverse professional figures (administrative, clinical, operational) as well as different ways they interact with data in hospital's information systems.The General Hospital Manager, the Controller, the ED Clinical Director and the COU (Complex Operating Unit) Clinical Director have been selected for the interviews.None of the interviewees mentioned anything that could be used to identify specific patients, clinical records, and/or the interviewees themselves.Interviews are therefore fully anonymous.As a consequence, for the correct interpretation of the assessment results, a general profile of each interview participant is presented in Table 1.
The semi-structured interviews have been conducted individually with each stakeholder in Table 1.Interviews started with a brief description of administrative data, the explanation of study aim and methodology, and the specification of the framework.Then, the participants have been asked to evaluate different aspects of the framework.In one case, the evaluation consisted in applying a binary scale (see S1 Table ); in another case, a five-point Likert scale ranging from 1 (highly irrelevant) to 5 (highly relevant) has been used (see S2 Table ).For the evaluation, the indicators of the original framework have been aggregated into groups based on the indicators' scope and dimension, such as "Quality patient-related metrics: Adverse events".To illustrate each group, some examples of the indicators belonging to the groups were given together with the questionnaire (see S2 Table ).

Framework's design
This section describes the results of the five steps of method's phase 1).

Administrative data
Administrative data are records on care delivery that are collected for management rather than research objectives [29].They are generated at the patient discharge from the hospital or hospital-based facility as part of standard coding procedures [30].Although databases vary in the design and data covered [31], they typically contain basic administrative information about the patient (e.g.age, gender, race, residence code, admission and discharge date), limited clinical data regarding hospital stay (e.g.patient's conditions, procedures received, vital signs, drugs prescribed), and financials (e.g.source of payment, cost of procedures) [32,33].Considering their nature, they present some non-trivial ethical challenges, particularly regarding their storage and accessibility [34].
In health services research, two major types of administrative data are used, namely claims data and hospital discharge abstracts [33].The former represents billing data to the insurer generated by providers for reimbursement purposes; the latter describes hospital-based services for inpatients abstracted from medical records.The corresponding data sources are called physician billing and hospitalization databases respectively [35].The main difference is that claims data files are limited to what is billed to the specific payer and provide information across a range of inpatient and outpatient providers as long as the person remains enrolled, while hospital discharge abstract data sets contain records on hospitalizations and services of patients within that hospital only, regardless of payer [33].This work investigates performance evaluation with administrative data of ED setting.It therefore considers hospital discharge abstract data; in particular, records from the hospital information system regarding services provided to patients in the ED department.

Defining framework characteristics and key attributes of measures
Performance assessment tools can be designed to provide both qualitative and quantitative outcomes.The most widespread assessment approach is the calculation of performance indicators, which represent "statistical devices" designed to highlight potential problems in care, usually by demonstrating the distance between the calculated value and a benchmark [36].The development and application of performance assessment tools differ depending on the stakeholders they aim to support [37].For instance, clinical safety and treatment efficiency indicators may be of higher priority for clinicians, while patient satisfaction could be a main concern of a hospital administrator [38].
The framework developed in this work takes the perspective of ED stakeholders: managers and physicians, who manage operations and are the decision-makers for care delivery organization and planning.Hence, it is designed to provide care delivery performance monitoring and to enable prompt identification of areas for improvement.Furthermore, considering how resource-constrained hospitals are, the framework is conceived so as its implementation and usage do not represent an additional financial burden and do not require extra resource utilization [24,39].
The selected performance indicators must possess technical and value attributes (Table 2).The technical attributes mean that the indicators must be timely & simple, costless, and available.The value attributes mean the indicators are also valid, general and scalable.Jointly, the two attribute types ensure that the indicators are easy to apply in real-life contexts and provide enough information for an holistic performance evaluation [24,36,40].
The use of administrative data sources guarantees that the indicators are costless and available (they are already being collected); the analysis of the academic literature, presented below, ensures validity; the remaining features (scalability, generality, timeliness, and simplicity) are assessed by the authors together with the interviewees.

Literature review
This section describes the review of the academic literature to identify the most appropriate metrics (indicators and measures) used for ED performance assessment following the criteria in Table 2.The review is based on the widely accepted protocols developed by Tranfield, Denyer, and Smart [41] and Greenhalgh [42].Scopus and Web of Science (WoS) have been used in combination to identify papers containing ED performance indicators and measures.The following search strings, "performance indicator*", "performance measure*", "emergency department", "ED" have been looked for in the papers' title, abstract and keywords.No time restrictions have been applied.An additional search has been made to capture those metrics using administrative data.The search strings in this case included, "administrative data", "claims", "routine data", and "performance".In both cases, the terms "overall", "global", "general", and "whole" have been added to capture generalpurpose indicators, as indicated in Table 2.Only peer-reviewed research papers and reviews written in English have been considered.
The screening process included the following steps (the resulting number of articles is provided in parentheses): citation extraction from the two databases (n = 108), elimination of duplicates (n = 22), title screening (n = 86), abstract screening (n = 80), and full-text analysis (n = 40).Moreover, each article was required to include metrics with the following additional criteria: • explicitly referred to as measuring performance; • designed or applied to ED setting; • evaluate overall performance; • not being specific to any clinical conditions (diseases or presenting complaints).Among the 40 selected articles, 24 fulfilled the additional criteria above.Those were selected for full-text analysis.At this point, the references of each of the 24 articles were scanned for relevant material.Three more articles have been included as a result.The final sample consists of 27 articles.

Emergency department performance framework
A total of 877 performance metrics have been identified in the 27 studies included in the fulltext review.Applying the selection criteria defined above, 224 metrics (measures and indicators) have been extracted and organized into the performance measurement framework presented in Fig 1.
The framework is designed as a cube composed of metrics' groups within the three related measurement dimensions: performance, analysis unit, and time-period.The performance dimension is represented by quality of care and time-efficiency [36,43,44].This classification is drawn from the primary function of an emergency department, as highlighted in the report Institute of Medicine [45]: the provision of high-quality care in a timely way.The analysis unit includes the entities in a performance evaluation procedure, namely physician, nurse, disease, triage category etc. as well as the overall facility level.The time-period allows to set timeframes of performance analysis, such as a month, a quarter, or a year.Overall, the analysis unit and time-period help in data grouping and filtering, while the performance dimension reveals the evaluation domain.
Performance metrics identified in the literature have been organized into five groups within the performance dimension.They are represented in the cells on the cube's top layer.Following performance dimension representation, the "temporal" and "quality" metrics have been specified, which in turn have been divided into patient-related and provider-related.This arrangement results in four performance metrics groups: temporal patient-related, temporal provider-related, quality patient-related, and quality provider-related.The quality metrics are those designed for evaluation of the care delivery process and patient outcomes and specified by numbers or proportions of favorable or unfavorable results.The temporal metrics are the waiting times concerning the patient journey and related operations, expressed in time units, such as hours and minutes.The throughput metrics have been placed at the intersection of these four groups.These metrics describe the ED's usage through the patient flow analysis.They do not represent ED performance directly as quality or temporal metrics, but provide information regarding the care process organization.By their nature, they can help reveal the causes of time inefficiencies and bad quality outcomes.In other words, they represent the connection between measurement and management [46].The cube's underlying idea is that a hospital stakeholder can choose a "slice" at the evaluation and time-period level of interest and calculate different types of performance metrics (e.g., yearly report of guideline adherence by individual physician).The choice depends on the goal of performance measurement and the desired detail.Therefore, the cube represents a flexible instrument that makes it possible to create targeted performance reports.The selected 224 performance metrics, which have been organized into the five performance groups defined above (Fig 1 ), are presented in Tables 3-7 below.Their description is reported in the original form.Table 3 represents the group of temporal patient-related metrics, where all 41 extracted performance metrics can be referred to as patients' length of stay (LOS) between various points of a care pathway, such as arrival, triage, and care provision.The 23 metrics representing the provider-related time group have been categorized in Decision-related times (12), Interventions and procedural times (8), and Triage time-efficiency (3), as shown in Table 4.The patient-related quality group is presented in Table 5.It is composed of 42 metrics representing the three types of patient outcomes, namely patient leaves (18), unplanned return visits within different timeframes (18), and adverse events (6).Table 6 shows provider-related quality performance metrics, which represent results of the care delivery and decisions made by medical personnel.It is the largest group with a total of 63 performance metrics, which have been categorized as Treatment Decisions: Procedures (21) and drugs (6), Documentation (21), and Guideline adherence (15).The first category represents decisions regarding performed procedures and drug administration; the second evaluates the patient data collection practice; and the last one helps to reveal the clinical practice behavior of healthcare personnel.
The last group is composed of throughput metrics, which represent counts of patients using the ED system (Table 7).The selected 55 metrics have been categorized into Patient flow by disposition (25): Admission (15), Transfer (8), and Discharge (2), Overall ED occupancy (24); and Patient flow by medical stuff (6).

Use-case example
The following example helps visualizing how the framework can be used in a real-world scenario.A hospital manager is asked by a director of one of his/her departments the reason as to why numerous patients with a specific clinical condition have recently visited ED and were discharged.Clearly, it is necessary to analyze the performance of the treatment of these patients in the ED.For example, one may want to study how the process was organized to understand if there is room for the implementation of a procedure to schedule a visit for such patients at discharge without affecting ED's overcrowding.In this case, the hospital manager may also want to avoid repetition of some costly exams by sending the diagnostic results directly to the clinician with which the follow up visit will be scheduled.However, the hospital does not collect a specific register for this group of patients; on the other hand, it has an administrative database that registers ED attendances.The starting point in the framework application is the study of administrative and clinical information presented in the database.The next step is the choice of performance indicators that can be calculated using the data at hand and are relevant for the investigation.Fig 2 shows an example of the cube for the described situation: it is sliced at the time-period "single year" and analysis unit "specific diagnosis" dimensions, with the Performance dimension on the front side containing corresponding performance metrics groups and the number of selected indicators in parenthesis.Such high-level representation highlights the dimensions along which performance can be monitored with the available data.At this point, the selected metrics should be calculated and analyzed, enabling the transition from performance measurement to performance management [46].

Experts' assessment
The framework developed in the previous section has been discussed through a number of semi-structured expert interviews.The main goal of these focus groups has been exploring the thoughts about the framework of key ED figures of a teaching university hospital.All the interviewees are involved in the monitoring of ED performance.They are collectively referred to as decision stakeholders.Expert interview is a method of qualitative empirical research to detect expert assessment about a topic that has been considerably used for knowledge production.Compared to multiple experiments, it allows to shorten time-consuming data gathering processes, particularly if the experts possess much practical tacit knowledge [71].It also serves as an alternative to running in situ experiments, which, particularly in the healthcare sector, might result in dire consequences in case of failures.
The results for the questions regarding the awareness of administrative data importance and an overall evaluation of the framework are presented in Fig 3 (S1 Table).The General Hospital Manager is the only stakeholder not fully aware that administrative data can be used for performance assessment.All stakeholders agreed with the proposed clustering of the indicators; they also share the idea that such structure helps evaluating the results.The framework has been judged unanimously as comprehensive for multidimensional performance evaluation and useful to support professional activity, and described as "clear and intuitive", "useful for planning", able to "reveal inefficiencies in care process" and "transform existing data into decision support information".
With respect to ease of implementation in practice, the opinions are different.The Controller and ED Clinical Director indicate that the correct implementation requires effort in terms of time and bureaucratic procedures' implementation; therefore, they think the framework is not easy to implement.On the contrary, the General Hospital Manager and COU Clinical Director have a more positive attitude about the framework's implementation.During the sessions, only the Controller has replied that no additional collection and digitalization of information on patient visits would be required to measure the performance with the proposed framework.
As a further step, to demonstrate how the framework can be adapted to the needs of the final user(s), participants have been asked to evaluate the framework's content with respect to two dimensions, namely motivation and decision-making (S2 Table ).The first one refers to the impact that each indicator has on a personal system of incentives toward professional goals.The last one evaluates the usefulness of information provided by an indicator(s) from a specific group for the management of operations (planning, monitoring, control, etc.) and working context analysis.
The results of the evaluation by each stakeholder are presented in Fig 4 .Some indicator groups have been given contradictory evaluations, as in the case of 4.1 (Quality providerrelated metrics: Treatment decisions regarding procedures, such as CT scans, X-ray counts, lab studies), which has a low importance for the COU Clinical Director and is of high interest for the General Hospital Manager.This is an expected outcome as each indicator group must be considered within its scope of application as well as its relevance for the goals of each stakeholder.In this case, the indicators in the evaluated group provide information that is relevant from the economic point of view, but is inconsequential for treatment decisions.
Focusing on the pattern of each stakeholder result, it can be noted that for most indicator groups the evaluation in both dimensions coincide, so they are placed on the diagonal.This demonstrates that the framework is relatively balanced as a control system because it impacts equally on the motivation to contribute towards corporate objectives and on decision making activities.However, in the evaluation made by the Controller, almost half of the indicator groups (7 out of 15) have been placed above the diagonal, meaning that the decision-making aspect prevails.This can be explained by the administrative-operational focus of its professional activity, which is based on daily data analysis.In addition, such bidimensional evaluation mapping can help in the identification of "core indicators" for each stakeholder.In the present analysis, with a scale from 1 to 5, the most relevant indicator groups are those evaluated 3 or higher in both dimensions (the upper right quadrant in blue in Fig 4).

Discussion
Little attention has been given so far to the usage of administrative data in emergency departments (EDs).On the one hand, for ED functions, performance measurement is traditionally focused on processes or outcomes (e.g., triage, diagnostics, treatment and disposition of patients) and time targets [15,16].On the other hand, as lamented by Madsen et al. [15], performance measurement systems in EDs tend to be narrowly focused on small sets of indicators designed for a specific target (e.g., resuscitation rate).Recently, Wachtel and Elalouf [1] have argued that for EDs too, performance must be measured in a holistic, multi-dimensional way.
They have proposed to include also non-clinical variables to monitor the performance of EDs (in Wachtel and Elalouf's [1] case, the number of accompanying escort has been found to be a significant predictor of a patient's LOS).This work follows Wachtel and Elalouf's [1] approach.In particular, this research designs a framework to harness the value of routinely produced records of administrative data to improve performance management of EDs and health systems in general [1,20,72,73]; 2).This is particularly useful as the adoption of this ED performance measurement framework will contribute to the use of underexploited, already available health care data [18].By leveraging routinely registered records, the framework supports tailored operational decision-making at various organisational levels, resulting in supporting ED management.This is relevant for EDs, which are heavily constrained in terms of time and resources [1,2].
To be used for measuring performance, rather than registering events, administrative data need to be arranged in a way that makes their information relevant.This is one of the framework's major contributions.Clearly, administrative data could be arranged in many different ways.Starting from the classic Donabedian's [74] "structure", "process", and "outcome" model for the assessment of the quality of care, to the multiple dimensions (e.g., appropriateness, expenditure, governance, improved care, clinical focus, efficiency, safety, sustainability and timeliness) identified by, among others, Zaadoud, Chbab, and Chaouch [44], Grimmer et al. [75], and Arah et al. [76], many different frameworks focused on patients' clinical data have been proposed.
The framework in this research abides by the following logic.The core dimension of the framework is evidently performance, as it defines the interpretation of the metrics calculation, while the other two dimensions (analysis unit and time-period) represent their selection bounds.Specifically, performance is measured by taking into account both the analysis unit and the time-period.This three-fold grouping in Fig 1 clusters unorganized administrative data in a way that they can be used for performance measurement because each group produces an unambiguous outcome [36].In fact, this structure allows for slicing at the desired levels as well as for the creation of metrics sets that allow detailed performance evaluation and reporting (e.g., the monthly throughput for a certain disease).To further improve the framework's ability to provide meaningful results, the selected performance metrics have been organized into multi-level groups, in Tables 3-7, that further narrows down their scope and make them more precise.Finally, the indicators have been selected from the literature.This ensures that each of them is relevant to measure a particular performance aspect [43].Nonetheless, indicators have been retained only if they were also timely, simple, costless, and available [44].The rationale is that the framework is conceived to exploit already available data-administrative data-without increasing the burden on a heavily resource-constrained system [24,39].
Expert interviews have been performed to obtain feedback on the framework.Expert discussion has been paramount to ensure that the framework actually fulfills its role, i.e., it provides on field decision support [28,43].During these interviews, the framework has been recognized as a useful tool for performance monitoring and managing operations.Moreover, the interviews showed that the stakeholders' professional role and objectives affect the indicators' perceived importance (Fig 4).This finding is in line with existing literature, which stresses the importance not only of including different indicators, but also of encompassing the perspective of various stakeholders on the same indicators to achieve a comprehensive performance evaluation [77].
Overall, the framework represents a practical tool designed for fast, regular, balanced, and systematic internal ED performance measurement and control, whereby professionals with different roles can select the subsets of indicators mostly tailored to their diverse needs [24,78].The framework therefore supports decision-making with respect to specific interests, responsibilities and objectives at each level of the organization.This contribution timely addresses recognized difficulties of decision-makers to choose a performance measurement system that would suit their needs without increasing their organization's financial burden [79].

Conclusions and limitations
Administrative data can be a new cornerstone for health care operation management.Using the existing information systems for decision-making support is essential in the resource-constrained hospital environment.This paper proposes a practical framework for performance analysis of hospital emergency departments based on administrative health records.It is aimed to assist decision stakeholders in regular and systematic control of ED performance at the desired level of details in terms of analysis unit, time period and performance dimension.The flexible design allows to identify the core indicators for each target user with respect to his/her professional role and objectives.As a mean of preliminary validation, the framework has been discussed with key ED decision stakeholders of a teaching hospital and it has been judged as comprehensive and useful for managing operations.The current work is a starting point for ED stakeholders to exploit the available administrative data sources to derive valuable performance information for decision-making.In fact, the framework could provide a blueprinting for more advanced, data-driven service design applications, such as BPMN-based approaches for the reengineering of healthcare processes [80,81].
This research has some limitations.First, in the choice of the metrics to include in the framework only one possible type of administrative data has been considered: the records on care delivery.This implies that the selected indicators refer to care delivery processes and outcomes.At the same time, other types of administrative data sources may be used to evaluate additional performance dimensions, such as structural (attributes of the care settings: equipment, resources etc.) or financial (costs) dimensions [15,82].Second, the included metrics have been selected to be computationally simple, as the framework is designed as a practical screening tool to be used by hospital decision makers for performance monitoring and control without any further resource needed.Without this constrain, it would evidently be possible to include other relevant indicators.

Table 2 . The key attributes of performance indicators. Attribute type Attribute Description
GeneralThe indicators evaluate the overall ED performance, rather than a specific clinical condition;ScalableThe indicators allow internal monitoring as well as comparison between EDs https://doi.org/10.1371/journal.pone.0293401.t002

Table 3 . Temporal patient-related metrics.
a The frequency is higher than the number of references as some authors discussed the separated metrics that we merged into one https://doi.org/10.1371/journal.pone.0293401.t003